On-Line Learning of Undirected Sparse n-grams

n-grams are simple learning models considered state-of-the-art in many sequential domains. They suffer from an exponential number of parameters in their width, n. We introduce undirected sparse n-grams, which store probability estimates only for some n-tuples from the unconditional joint distribution, specifically the most frequent. Experimental results show this sparse version can outperform a narrower n-gram with the same number of parameters. In addition, our models predict equally well forward, backward, or in any combination. We present an on-line learning algorithm which induces both model parameters and structure (which patterns to include), despite a priori uncertainty about which patterns are frequent. Dynamic pattern inclusion complicates probability estimation during on-line learning, but we present and examine several solutions, including a Bayesian approach. Lastly, we describe multiwidth combinations of sparse n-grams that solve two important problems with the simple sparse models, and which are useful for hierarchical identification and composition of repeated substructure in data. These hierarchical sparse n-grams can be learned on-line with no fixed bound on width, using less memory than that taken by the training data.